Since AlphaGo, AI researchers have recognized the promise of integrating reinforcement learning with search methods, which involve considering many potential next actions available to an RL agent, and simulating what their results might be before choosing one. This starts to mimic human deliberation much more closely, by explicitly introducing elements of "planning" into the RL paradigm. Yang attributes the huge performance improvements of AlphaGo, AlphaZero and MuZero to this search process. Another important distinction in RL is between model-based systems, which construct explicit models of their environments, and model-free systems, which don't. Prior to AlphaGo, just about all leading RL work was done on model-free systems (PPO and deep Q learning, for example). Model-based systems just weren't practical because the learning environment models is hard, and adds a significant layer of complexity on top of the simpler action selection task that model-free systems could focus on exclusively.

The automotive industry was the first to promote the development of applications of model-based systems technology on a broad scale and, as a result, has produced some of the most advanced prototypes and products. In this article, we illustrate the features and benefits of model-based systems and qualitative modeling by prototypes and application systems that were developed in the automotive industry to support on-board diagnosis, design for diagnosability, and failure modes and effects analysis. Car manufacturers and their suppliers face increasingly serious challenges particularly related to fault analysis and diagnosis during the life cycle of their products. On the one hand, the complexity and sophistication of vehicles is growing, so it is becoming harder to predict interactions between vehicle systems, especially when failures occur. On the other hand, legal regulations and the demand for safety impose strong requirements on the detection and identification of faults and the prevention of their effects on the environment or dangerous situations for passengers and other people.

There is much evidence that humans and other animals utilize a combination of model-based and model-free RL methods. Although it has been proposed that these systems may dominate according to their relative statistical efficiency in different circumstances, there is little specific evidence -- especially in humans -- as to the details of this trade-off. Accordingly, we examine the relative performance of different RL approaches under situations in which the statistics of reward are differentially noisy and volatile. Using theory and simulation, we show that model-free TD learning is relatively most disadvantaged in cases of high volatility and low noise. We present data from a decision-making experiment manipulating these parameters, showing that humans shift learning strategies in accord with these predictions. The statistical circumstances favoring model-based RL are also those that promote a high learning rate, which helps explain why, in psychology, the distinction between these strategies is traditionally conceived in terms of rule-based vs. incremental learning.

The automotive industry was the first to promote the development of applications of model-based systems technology on a broad scale and, as a result, has produced some of the most advanced prototypes and products. In this article, we illustrate the features and benefits of model-based systems and qualitative modeling by prototypes and application systems that were developed in the automotive industry to support on-board diagnosis, design for diagnosability, and failure modes and effects analysis.

The automotive industry was the first to promote the development of applications of model-based systems technology on a broad scale and, as a result, has produced some of the most advanced prototypes and products. In this article, we illustrate the features and benefits of model-based systems and qualitative modeling by prototypes and application systems that were developed in the automotive industry to support on-board diagnosis, design for diagnosability, and failure modes and effects analysis.